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Introduction 

Hsiang  et  al.  (2014;  henceforth  HBM)  discuss  five  alleged  errors  committed  by  Buhaug  et 
al.  (2014).  We  assess  these  claims  and  find  that  they  largely  miss  the  target.  HBM  are  correct 
in  pointing  to  a  minor  coding  error  and  we  also  agree  with  HBM  that  Buhaug  et  al.  could 
have  been  clearer  in  explaining  some  of  their  proposed  modifications.  Those  issues 
notwithstanding,  we  conclude  that  Buhaug  et  al.’s  concerns  about  the  viability  of  the  fixed 
effect-based  meta-analysis  presented  in  Hsiang  et  al.  (2013)  remain  valid. 


On  meta-analysis 

The  main  disagreement  between  Hsiang  et  al.  (2013)  and  Buhaug  et  al.  (2014)  concerns 
which  parameter  to  rely  on  when  using  meta-analysis  to  make  inferences  about  the  general 
relationship  between  climate  and  conflict.1  A  brief  discussion  of  some  facets  of  meta¬ 
analysis  is  in  order.  As  Gelman  et  al.  (2014,  p.  125)  explain,  the  researcher  may  be  faced 
with  three  kinds  of  samples: 

The  first  possibility  is  that  we  view  the  studies  as  identical  replications  of  each  other,  in  the 
sense  that  we  regard  the  individuals  in  all  the  studies  as  independent  samples  from  a 
common  population,  with  the  same  outcome  and  so  on.  A  second  possibility  is  that  the 
studies  are  so  different  that  the  results  of  any  one  study  provide  no  information  about  the 
results  of  any  of  the  others.  A  third,  more  general,  possibility  is  that  we  regard  the  studies 
as  exchangeable  but  not  necessarily  either  identical  or  completely  unrelated. 

The  model  normally  used  when  one  believes  the  data  correspond  to  the  first  possibility  is 
the  so-called  fixed  effect  model,2  whereas  a  random  effects  model  is  appropriate  if  one  believes 
the  data  resemble  the  third  possibility  (Borenstein  2011).  The  second  possibility  described 
by  Gelman  et  al.  does  not  have  a  model  as  it  does  not  make  sense  to  conduct  a  meta-analysis 
of  disparate  and  unconnected  studies. 

Having  chosen  a  random  effects  model,  there  are  two  different  quantities  one  may  sensibly 
report,  the  mean  of  the  distribution  of  effect  sizes,  n  (and  its  estimated  standard  error),  and 
the  predicted  distribution  of  effects,  9.  ju  will  have  a  wider  standard  error  than  the  estimated 
mean  effect  in  a  fixed  effect  model  because  the  random  effects  model  takes  into  account 
between-study  precision  as  well  as  within-study  error.  The  wider  the  spread  of  effects  in  the 
meta-study  sample,  the  more  weight  is  put  into  the  individual  study  estimates,  while  the 
larger  the  error  in  individual  studies,  the  less  weight  is  put  into  that  individual  estimate. 


1  Buhaug  et  al.  also  raise  a  second  concern;  the  likelihood  of  selection  bias  in  Hsiang  et  al.  (20 1  3)  by  (i)  suppressing 
studies  of  previously  tested  associations,  (ii)  only  selecting  one  climate  indicator  from  each  sample  study,  and  (iii) 
not  accounting  for  variations  in  potential  for  generalization  between  studies.  Since  HBM  did  not  respond  to  this 
concern,  beyond  conducting  a  hypothetical  “stress  test”  based  on  the  inappropriate  fixed  effect  model,  we  do 
not  revisit  this  point  here. 

2  Not  to  be  confused  with  fixed  effects  regression  models,  which  may  be  used  in  individual  sample  studies. 
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If  one  is  confident  that  the  sample  studies  are  estimating  mosdy  the  same  effect,  reporting 
[i  makes  sense.  If,  on  the  other  hand,  one  believes  the  effects  are  likely  to  be  different  (but 
not  so  different  that  a  meta-analysis  carries  no  meaning),  one  should  report  and  interpret 
the  more  conservative  9: 

Uncertainty  about  the  probable  treatment  effect  in  a  particular  population  where  a  study 
has  not  been  performed  (or  indeed  in  a  previously  studied  population  but  with  a  slightly 
modified  treatment)  might  be  more  reasonably  represented  by  inference  for  a  new  study 
effect  [6],  exchangeable  with  those  for  which  studies  have  been  performed,  rather  than  for 
the  overall  mean  [/<].  (Gelman  et  al.  2014,  pp.  127-8;  terms  in  brackets  added) 

It  should  be  clear  that  deciding  on  the  model  to  estimate  and  the  parameter(s)  from  the 
model  to  report  depends  on  the  assumptions  about  the  underlying  data,  whether  they 
represent  one  effect,  more  or  less  the  same  effect,  or  mostly  different  effects.  This  decision 
has  fundamental  implications  for  the  goal  of  the  meta-analysis:  At  one  extreme  (the  fixed 
effect  method),  the  goal  is  to  estimate  the  true  effect  size  -  because  one  believes  there  is 
such  a  thing.  At  the  other  extreme  (i.e.,  reporting  the  conservative  9  from  a  random  effects 
model),  the  goal  is  to  estimate  the  true  distribution  of  effects  that  gave  rise  to  our  sample 
effects  (or  alternatively  estimate  the  predicted  effect  of  a  hypothetical  future  study).  If  this 
distribution  is  approximately  normal,  then  it  is  sensible  to  report  the  2.5th,  50th  (the  median) 
and  97.5th  percentile  from  the  estimated  distribution  of  the  random  effects  model. 

Against  this  backdrop,  we  assess  the  reply  by  HBM  to  Buhaug  et  al.  We  discuss  the  five 
alleged  errors  in  the  order  they  are  presented  in  HBM. 


1. 

Buhaug  et  al.’s  (2014)  modified  meta-analysis  reports  9  from  a  random  effects  model  (in 
this  case,  a  Bayesian  hierarchical  model).  HBM  criticize  this  modeling  choice  and  argue 
that  they  should  have  followed  Hsiang  et  al.  (2013)  in  reporting  the  estimated  mean  and 
confidence  interval  from  a  fixed  effect  model  instead.  However,  as  discussed  above,  the 
fixed  effect  approach  rests  on  a  strong  assumption  about  causal  homogeneity  which  clearly 
is  violated  in  this  case.  In  the  words  of  Buhaug  et  al.  (p.  3), 

[the]  sample  of  ‘intergroup  conflict’  studies  covers  a  wide  range  of  social  phenomena,  from 
non-violent  land  grabbing  via  urban  riots  to  major  civil  war;  a  wide  range  of  climatic  events, 
from  heat  waves  via  excess  rainfall  to  global  ENSO  cycles;  and  a  wide  range  of  spatial  scales, 
from  municipalities  via  countries  to  the  entire  world.3 

In  other  words,  the  data  obviously  are  closer  to  the  second  than  the  first  possibility  outlined 
by  Gelman  et  al.  in  the  quote  above,  rendering  the  fixed  effect  model  inappropriate.  Relying 
on  //  and  its  standard  error  also  requires  confidence  that  the  sample  studies  are  measuring 
mostly  the  same  causal  effect.  We  lack  such  confidence  and  believe  that  Buhaug  et  al.’s 


3  In  this  sense,  HBM’s  (pp.  3-4)  hypothetical  example  of  how  to  estimate  the  mean  age  for  a  sample  with  100 
observations  of  a  similar  kind  (individuals)  with  a  known  age  distribution  is  misguided;  the  data  considered  in 
Hsiang  et  al.  (20 1  3)  are  nothing  of  that  sort. 
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decision  -  to  report  9  from  a  streamlined  sample  of  civil  conflict  studies  -  is  the  most 
reasonable  approach.  Indeed,  in  their  supplementary  material  (2013,  p.  19),  Hsiang  et  al. 
acknowledge  that  causal  heterogeneity  is  a  likely  problem: 

It  is  likely  the  case  that  differences  among  estimated  effect  sizes  are  not  due  to  sampling 
variability  alone.  That  is,  studies  looking  at  the  effect  of  climate  on  different  outcomes  might 
be  expected  to  share  some  similarities  (different  outcomes  might  be  related),  but  also  some 
important  differences  (some  outcomes  or  samples  might  exhibit  different  responses  to 
climate). 

Hsiang  et  al.  (supplementary  material)  go  on  to  discuss  the  Bayesian  hierarchical  model 
and  they  also  visualize  9  in  the  right  panel  of  their  Fig.  5. 4  However,  they  give  undue 
emphasis  to  the  fixed  effect  model  when  concluding  on  the  strength  and  robustness  of  the 
general  association  between  climate  and  intergroup  conflict  (i.e.,  they  report  an  average 
effect  of  11.1%  (±  2.62%)  increase  in  intergroup  conflict  for  each  standard  deviation 
increase  in  climate). 


2. 

HBM  criticize  Buhaug  et  al.  of  presenting  a  figure  (Fig.  1)  that  is  not  directly  comparable 
to  Hsiang  et  al.  (2013)  as  it  is  based  on  a  model  that  is  specified  with  all  lagged  effects 
instead  of  relying  on  some  contemporaneous  and  some  lagged  effects  as  Hsiang  et  al.  do. 
The  reasoning  behind  Buhaug  et  al.’s  modeling  choice  is  partly  to  maximize  analytical 
consistency  and  partly  to  ensure  that  the  treatment  occurs  prior  to  the  outcome.  This  is 
fully  explained  in  Buhaug  et  al.  (2014,  p.4)  and  does  not  constitute  an  error  but  rather 
reflects  disagreement  with  the  model  specification  preferred  by  Hsiang  et  al. 

However,  we  note  that  Buhaug  et  al.  failed  to  explain  why  they  excluded  the  result  for 
Hsiang  et  al.  (2011)  from  Fig.  1.  The  reason  is  again  analytical  consistency;  The  Hsiang  et 
al.  (2011)  study  considers  a  global  ENSO  effect  whereas  the  studies  in  the  valid  sample 
investigate  a  local  effect  (i.e.,  the  treatment  is  measured  specifically  for  each  unit  of 
analysis).  Moreover,  Maystadt  et  al.  (2013)  was  excluded  because  Buhaug  et  al.  failed  to 
obtain  the  original  replication  data  for  the  (at  the  time)  unpublished  study  and  hence  were 
unable  to  re-estimate  their  model  with  lagged  climate  indicators.  This  is  stated  in  footnote 
5  in  Buhaug  et  al.’s  supplementary  information. 


3. 


HBM  criticize  Buhaug  et  al.  for  reclassifying  some  of  the  studies  described  in  Hsiang  et  al. 
(2013).  Specifically,  Buhaug  et  al.  exclude  the  effects  of  ENSO  and  PDSI  when  estimating 
the  aggregate  temperature  effect.  The  reason  is  simple:  ENSO  and  PDSI  do  not  measure 
temperature.  PDSI  is  a  measure  of  drought;  ENSO  is  a  fluctuating  climatological 


4  To  be  precise,  they  estimate  two  slightly  different  distributions,  one  derived  from  a  variance-weighted  random 
effects  model  and  one  using  the  Bayesian  hierarchical  model. 
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phenomenon  associated  with  shifts  in  sea  surface  temperature  and  atmospheric  pressure 
across  the  Equatorial  Pacific  Ocean,  with  different  local  manifestations.  Similarly,  Buhaug 
et  al.  decided  to  follow  convention  and  classify  the  Standardized  Precipitation  Index  (SPI) 
as  an  indicator  of  drought.  While  these  alterations  could  have  been  better  explained,  we  fail 
to  see  how  Buhaug  et  al.’s  more  conventional  classification  constitutes  an  error. 


4. 

HBM  identify  a  coding  error  in  Buhaug  et  al.,  whereby  one  of  the  two  results  from 
O’Loughlin  et  al.  (2012)  is  unaccounted  for  when  Q  for  the  aggregate  temperature  effect  is 
estimated.  This  is  a  correct  observation,  and  we  thank  HBM  for  pointing  this  out.  A 
comparison  of  the  original  and  corrected  figures  (Figs.  A  and  B)  reveals  that  this  error  does 
affect  the  shape  of  the  distribution  at  higher  values  but  has  little  influence  on  the 
interpretation  of  the  overall  temperature-conflict  pattern 


Fig.  A.  Original  estimates  for  the  effect  of  climate  variability  on  civil  conflict.  By  mistake,  the  temperature 
effect  of  O’Loughlin  et  al.  (20 1 2)  -  the  rightmost  red  plot  -  was  not  accounted  for  when  estimating  the  effect 
distribution  for  temperature  (red  curve).  See  Buhaug  et  al.  (201  3)  Fig.  I  for  further  details. 


%  change  per  1o  change  in  climate 
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Fig.  B.  Corrected  estimates  for  the  effect  of  climate  variability  on  civil  conflict. 


5. 

Lastly,  HBM  criticize  Buhaug  et  al.  for  not  visualizing  the  variance-weighted  mean  and 
confidence  interval  in  their  Fig.  1.  This  goes  back  to  the  discussion  in  Section  1  above.  To 
reiterate,  we  believe  the  fixed  effect-based  estimation  that  Hsiang  et  al.  (2013)  apply  is 
inappropriate  in  this  setting  given  the  large  heterogeneity  in  the  data  sample.  A 
visualization  of  the  distribution  of  values  from  a  random  effects  model  (e.g.,  the  Bayesian 
hierarchical  model)  and  interpretation  of  the  shape  and  range  of  the  effects  in  our  view 
make  more  sense  since  we  lack  a  solid  theoretical  foundation  to  expect  a  single,  common 
climate  effect. 


A  Comment  on  Hsiang  et  al.’s  Reply  to  Buhaug  et  al.  |  9 


Concluding  remarks 

We  thank  HBM  for  their  reply,  which  allowed  us  to  correct  a  minor  coding  error  and 
provide  a  more  detailed  explanation  of  the  concerns  raised  in  Buhaug  et  al.  (2014).  In  fact, 
we  acknowledge  that  we  may  not  have  given  Hsiang,  Burke,  and  Miguel  the  credit  they 
deserve  for  embarking  on  a  truly  innovative  project,  with  the  aim  of  providing  a  more 
transparent  and  rigorous  quantification  of  the  state-of-the-art  on  climate  and  conflict  than 
extant  qualitative  literature  reviews  have  been  able  to  offer.  We  also  appreciate  their 
exemplary  replication  material,  which  was  easy  to  use  and  read. 

So,  where  do  we  stand  on  this?  The  following  brief  statements  may  be  illuminating: 

a)  We  agree  that  meta-analysis  in  principle  can  be  a  useful  tool  to  synthesize  and 
quantify  results  from  the  empirical  climate-conflict  literature  -  provided  the 
researcher  investigates  comparable  studies  of  similar  phenomena  and  accounts  for 
variations  in  representativeness  and  potential  for  generalization.  This  is  why 
Buhaug  et  al.’s  modified  meta-analysis  was  run  on  a  reduced  sample,  limited  to  civil 
conflicts  (although  one  might  credibly  argue  that  the  sample  should  have  been 
streamlined  further  by  also  limiting  focus  to  a  singe  type  of  climate  parameter  and 
involving  one  type  of  actor  only). 

b)  We  do  not  believe  in  a  single,  common  climate  effect  that  connects  various  distinct 
climatic  conditions  (drought,  heatwave,  excess  rainfall,  etc.)  with  various  distinct 
forms  of  violent  conflict  (urban  riots,  land  grabbing,  insurgency,  etc.)  at  various 
spatiotemporal  scales  in  a  direct  and  simple  manner.  Lack  of  a  comprehensive 
theoretical  framework,  a  demonstrably  large  variation  in  estimated  effects  across 
studies,  and  careful  reading  of  the  empirical  literature  (e.g.,  Adger  et  al.  2014;  Klomp 
and  Bulte  2013),  also  are  at  odds  with  the  assumption  of  causal  homogeneity. 

c)  For  reasons  explained  in  b),  we  believe  a  meta-analysis  of  the  empirical  climate- 
conflict  literature  based  on  the  variance-weighted  fixed  effect  model,  where  the  goal 
is  to  estimate  the  “true”  climate  effect,  is  inappropriate.  We  also  believe  the  random 
effects-generated  mean  estimate  (ju)  in  itself  has  limited  value  in  this  context  since 
it,  too,  rests  on  quite  strong  assumptions  with  respect  to  causal  homogeneity. 

d)  We  agree  that  the  mass  of  effects  in  the  complete  pool  of  estimated  effects  of  climate- 
related  variables  on  civil  conflict  appears  to  be  larger  above  zero  than  below  zero;  in 
other  words,  more  than  half  of  the  area  of  0  in  Buhaug  et  al.’s  Fig.  1  is  located  above 
zero.  However,  this  is  different  from  concluding  on  a  sweeping,  positive  climate- 
conflict  relationship;  the  between-study  variation  simply  is  too  large,  rendering 
reference  to  (and  quantification  of)  a  singular  effect  of  little  substantive  meaning. 

e)  We  urge  scientists  to  exercise  caution  when  reflecting  on  the  wider  implications  of 
this  research.  The  tendency  toward  a  larger  mass  of  positive  effects  cannot  be  taken 
as  evidence  that  sustained  warming  will  increase  civil  conflict  risk  in  the  future.  That 
question  is  not  addressed  in  the  surveyed  literature,  which  studies  effects  of  climatic 
variability,  not  climate  change.  It  is  not  given  that  societies  tomorrow  will  respond 
similarly  to  what  will  then  be  normal  weather  as  societies  today  respond  to  climatic 
anomalies. 
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f)  We  believe  there  is  more  to  be  said  about  the  risk  of  selection  bias  than  what  we 
have  commented  on  here.  Specifically,  decisions  on  exact  operationalization  of 
climate  (and  conflict),  specification  of  functional  form  and  time  lag  of  the  effect,  and 
other  modeling  choices  may  be  informed  partly  by  the  author’s  post  hoc  knowledge 
about  their  statistical  implications.  Sampling  results  based  on  what  is  reported  in 
individual  studies  instead  of  specifying  a  coherent,  theory-informed  meta-analysis 
building  on  the  original  datasets  may  potentially  accentuate  such  bias.  This  concern, 
as  well  as  the  related  “file  drawer  problem”,  is  of  course  not  limited  to  climate  and 
conflict,  or  to  quantitative  syntheses  of  that  literature,  but  climate  change-related 
research  (and  funding  opportunities)  may  be  subject  to  particularly  powerful  norms 
of  conformity  that  reward  certain  results  more  than  others  (e.g.,  Lahsen  2013;  see 
also  Franco  et  al.  2014;  Kicinski  2013;  Rothstein  et  al.  2005;  Rosenthal  1979; 
Schooler  2011). 
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Last  year,  a  meta-analysis  published  in  Science  by 
Hsiang,  Burke,  and  Miguel  (HBM)  argued  that 
climate  is  robustly  and  causally  related  to  many 
forms  of  conflict.  A  commentary  article  by 
Buhaug  et  al.  questioned  the  assumptions  and 
sample  selection  strategy  underlying  this  analysis, 
and  showed  that  these  decisions  had  a  dramatic 
impact  on  their  conclusion.  In  a  reply  to  the  com¬ 
mentary,  HBM  dismiss  these  concerns  and  in¬ 
stead  claim  that  Buhaug  et  al.  committed  five  key 
errors  in  their  own  analysis.  We  assess  these 
claims  and  find  that  they  largely  miss  the  target. 
HBM  are  correct  in  pointing  to  a  minor  coding  er¬ 
ror  and  we  also  agree  with  HBM  that  Buhaug  et 
al.  could  have  been  clearer  in  explaining  some  of 
their  proposed  modifications.  Those  issues  not¬ 


withstanding,  we  argue  that  Buhaug  et  al.’s  con¬ 
cerns  about  the  viability  of  the  fixed  effect-based 
meta-analysis  presented  in  Hsiang  et  al.  (2013)  re¬ 
main  valid. 

While  we  agree  that  meta-analysis  in  principle  can 
be  a  useful  tool  to  synthesize  results  from  the  em¬ 
pirical  climate-conflict  literature,  we  find  it  im¬ 
plausible  to  assume  a  single  “true”  effect  that  con¬ 
nects  various  distinct  climatic  conditions  with  var¬ 
ious  distinct  forms  of  conflict  at  various  spatio- 
temporal  scales  in  a  simple  and  direct  manner. 

For  this  reason,  the  estimated  average  effect  that 
Hsiang  et  al.  refer  to  has  little  substantive  mean¬ 
ing.  Instead,  we  find  it  useful  to  interpret  the  dis¬ 
tribution  of  estimated  effects  across  studies, 
whose  variation  is  inconsistent  with  the  claim  of  a 
robust  aggregate  climate-conflict  relationship. 
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